iT邦幫忙

2023 iThome 鐵人賽
DAY 14
0
自我挑戰組

Techschool Goalng Backend Master Class 的學習記錄系列 第 14 篇

[Day 14] Understand isolation levels & read phenomena in PostgreSQL

15th鐵人賽 postgresql isolation level transaction golang
643 瀏覽

  • 分享至 

  • xImage
    •  

Isolation Levels in Postgres

  • 建立**Postgres** Container 與 simple_bank database.

    docker run --name postgres -p 5432:5432 -e POSTGRES_USER=root -e POSTGRES_PASSWORD=secret --network bank-network -d postgres:14-alpine
docker exec -it postgres createdb --username=root --owner=root simple_bank

  • Connect to Postgres and access simple_bankdatabase

    docker exec -it postgres psql -U root simple_bank

  • Get current isolation level in Postgres

    • **Postgres**預設都是Read committed
    • show transaction isolation level; 這個SQL指令時，所獲得的隔離級別是針對當前的console session Level**Repeatable Read**
    • Postgres 無法修改Global Session ****isolation level 
    simple_bank=# show transaction isolation level;
 transaction_isolation
-----------------------
 read committed
(1 row)

  • Change isolation level in Postgres

    • 在MySQL我們在建立transactions前可以改變isolation level
    • 但是Postgres 只有建立了transactions 才可以改變isolation level 
    -- Tx1:
simple_bank> begin;
BEGIN

simple_bank> set transaction isolation level read uncommitted;
SET

simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 read uncommitted
(1 row)

  • 建立accounts、entries、transfers Table

    CREATE TABLE "accounts" (
  "id" bigserial PRIMARY KEY,
  "owner" varchar NOT NULL,
  "balance" bigint NOT NULL,
  "currency" varchar NOT NULL,
  "created_at" timestamptz NOT NULL DEFAULT (now())
);

CREATE TABLE "entries" (
  "id" bigserial PRIMARY KEY,
  "account_id" bigint NOT NULL,
  "amount" bigint NOT NULL,
  "created_at" timestamptz NOT NULL DEFAULT (now())
);

CREATE TABLE "transfers" (
  "id" bigserial PRIMARY KEY,
  "from_account_id" bigint NOT NULL,
  "to_account_id" bigint NOT NULL,
  "amount" bigint NOT NULL,
  "created_at" timestamptz NOT NULL DEFAULT (now())
);

CREATE INDEX ON "accounts" ("owner");

CREATE UNIQUE INDEX ON "accounts" ("owner", "currency");

CREATE INDEX ON "entries" ("account_id");

CREATE INDEX ON "transfers" ("from_account_id");

CREATE INDEX ON "transfers" ("to_account_id");

CREATE INDEX ON "transfers" ("from_account_id", "to_account_id");

COMMENT ON COLUMN "entries"."amount" IS 'can be negative or positive';

COMMENT ON COLUMN "transfers"."amount" IS 'must be positive';

ALTER TABLE "entries" ADD FOREIGN KEY ("account_id") REFERENCES "accounts" ("id");

ALTER TABLE "transfers" ADD FOREIGN KEY ("from_account_id") REFERENCES "accounts" ("id");

ALTER TABLE "transfers" ADD FOREIGN KEY ("to_account_id") REFERENCES "accounts" ("id");

  • 插入三筆Account的資訊進到accounts

    INSERT INTO "accounts" ("owner", "balance", "currency") VALUES
('one', 100, 'USD'),
('two', 100, 'USD'),
('three', 100, 'USD');

-- Tx1:

simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

Read uncommitted isolation level in Postgres

  • 建立兩個Console與Transaction，並設定兩個Session的Isolation Level為**Read Uncommitted**

    -- Tx1:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level read uncommitted;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 read uncommitted
(1 row)

-- Tx2:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level read uncommitted;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 read uncommitted
(1 row)

  • 在Tx1 中對accounts table進行select query

    -- Tx1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 在Tx2 中對accounts table進行id=1的 query

    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |     100 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

  • 這時候在Tx1 中對Account 1的balance減去10

    -- Tx1
simple_bank=*# update accounts set balance = balance - 10 where id = 1 returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

UPDATE 1

  • 我們在Tx2 再次對Account 1進行Query

    • 雖然使用了**Read uncommitted**隔離級別，但結果仍然是100美元。
    • Postgres的文檔指出其**Read uncommitted與Read committed**行為相同。
    • 實際上，Postgres只有三個隔離級別，且最低級別是**Read committed**。
    • 一般情況下，**Read uncommitted**隔離級別並不建議使用。
    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |     100 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

  • 對Tx1 進行 commit 後，我們在Tx2 再次查詢Account 1，這時就可以看見Balance的修改

    --Tx1
simple_bank=*# commit;
COMMIT

--Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)
simple_bank=*# commit;
COMMIT

Read committed isolation level in Postgres

  • 建立兩個Console和Transaction，並設定兩個Session的Isolation Level為**Read Committed**

    -- Tx1:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level read committed;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 read committed
(1 row)

-- Tx2:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level read committed;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 read committed
(1 row)

  • 在Tx1 中對accounts table進行select query

    -- Tx1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 在Tx2 中對accounts table進行id=1的查詢並進行balance≥90的Query

    • 除了dirty read現象外，還希望了解phantom read 的處理方式。
    • 查找餘額大於或等於90美元的帳戶。
    • 有3筆記錄滿足此搜尋條件。
    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

simple_bank=*# select * from accounts where balance >= 90;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 這時候在Tx1 中對Account 1的balance減去10

    -- Tx1
simple_bank=*# update accounts set balance = balance - 10 where id = 1 returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

UPDATE 1

simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

  • 我們在Tx2 再次對Account 1進行Query

    • 這時Tx1 尚未committed ，所以Tx2 的Account 1 Query 依舊是90
    • 因此read-committed 可以避免dirty read 
    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      90 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

  • 將Tx1 進行commit 並在Tx2 再次對Account 1進行Query

    • 這時可以看見Tx2 中的balance已經被更新
    • 所以證明出現了non-repeatable 
    -- Tx1
simple_bank=*# commit;
COMMIT

-- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

  • 我們在Tx2 再次進行balance ≥90的Query

    • phantom read
      同樣我們在Tx2 進行同一個Query取得balance 90以上的Accounts，可以發現相同的Query但是Return 的Record 數量不相同，這就被稱為phantom read

      -- Tx2
simple_bank=*# select * from accounts where balance >= 90;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
(2 rows)

simple_bank=*# commit;
COMMIT

  • 所以read-committed 只能避免dirty read 無法避開non-repeatable 、phantom read

Repeatable read isolation level in Postgres

  • 建立兩個Console和Transaction，並設定兩個Session的Isolation Level為**Repeatable**

    -- Tx1:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level repeatable read;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 repeatable read
(1 row)

-- Tx2:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level repeatable read;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 repeatable read
(1 row)

  • 在Tx1 中對accounts table進行select query

    -- Tx1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 在Tx2 中對accounts table進行id=1的查詢與balance ≥80 的Query

    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

simple_bank=*# select * from accounts where balance >= 80;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 這時候在Tx1 中對Account 1的balance減去10

    • 我們知道在隔離級別「read committed」中已經阻止了dirty read , 所以就不需要再驗證
    -- Tx1
simple_bank=*# update accounts set balance = balance - 10 where id = 1 returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

UPDATE 1

simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

simple_bank=*# commit;
COMMIT

  • 我們在Tx2 再次對Account 1進行查詢，來確認是否可以獲得Tx1 所更新的balance

    • 這裡可以發現select query return Account 1的舊版本，balance為80美元，儘管Tx1已經將其更改為70並且成功commit。
    • 這是因為repeatable-read isolation level 確保所有的讀取查詢都是repeatable，這意味著，它總是返回相同的結果，即使其他已提交的交易進行了更改 (避免non-repeatable)。
    • 在進行balance ≥80 的Query 返回的Record數量依舊相同 (避免phantom read)。
    -- Tx2
simple_bank=*# select * from accounts where id = 1;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(1 row)

simple_bank=*# select * from accounts where balance >= 80;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      80 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

  • 如果我們在Tx2 對Account 1 的balance那結果會是如何?

    • 在MySQL的可重複讀取隔離級別中，允許餘額更新為60美元。
    • Postgres在遇到同時更新時出現錯誤。
    • 拋出該錯誤比讓餘額發生混淆的修改更為合理。
    • Postgres在這種情境下的表現值得讚揚。
    simple_bank=*# update accounts set balance = balance - 10 where id = 1 returning *;
ERROR:  could not serialize access due to concurrent update

simple_bank=!# rollback;
ROLLBACK

Serialization anomaly in Postgres

  • 建立兩個Console和Transaction，並設定兩個Session的Isolation Level為**Repeatable**

    -- Tx1:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level repeatable read;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 repeatable read
(1 row)

-- Tx2:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level repeatable read;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 repeatable read
(1 row)

  • 現在想像我們有一個使用情境，需要計算所有帳戶的餘額總和，然後用該總餘額建立一個新的帳戶

    -- Tx1:
simple_bank=*# select sum(balance) from accounts;
 sum
-----
 270
(1 row)

  • 總合為270美元，然後我們在帳戶表中插入一個新的記錄，其中擁有者為"sum"，餘額為270，且貨幣為"USD"。

    -- Tx1:

simple_bank=*# insert into accounts(owner, balance, currency) values ('sum', 270, 'USD') returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
(1 row)

INSERT 0 1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
(4 rows)

  • 但是如果也想在Tx2 中對accounts table進行這項操作呢?

    • 由於我們使用的是repeatable-read隔離級別，Tx2中的select查詢只會看到原始的帳戶列表，而不會看到交易1剛剛插入的新記錄。
    • 但我們一樣進行Sum 並做Insert時會發現lock
    -- Tx2
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
(3 rows)

simple_bank=*# select sum(balance) from accounts;
 sum
-----
 270
(1 row)

simple_bank=*# insert into accounts(owner, balance, currency) values ('sum', 270, 'USD') returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
(1 row)

  • 我們再對Tx1 和Tx2進行Commit，這時Tx2 的Select會出現在duplicate sum records

    • 這個現像稱為**Serialization anomaly**
    • 因為如果這兩個交易是**Serialization**運行的，一個接一個，那麼我們不可能有兩條總和為270的記錄。
    • 不管Tx1或Tx2先運行，我們應該有一條270美元的記錄，和另一條540美元的記錄。
    -- Tx1
simple_bank=*# commit;
COMMIT

-- Tx2
simple_bank=*# commit;
COMMIT

simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
  5 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00

(5 rows)

  • 所以Postgres 在repeatable-read隔離級別中無法避免**Serialization anomaly**

Serializable isolation level in Postgres

  • 建立兩個Console和Transaction，並設定兩個Session的Isolation Level為Serializable

    -- Tx1:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level serializable;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 serializable
(1 row)

-- Tx2:
simple_bank=# begin;
BEGIN
simple_bank=*# set transaction isolation level serializable;
SET
simple_bank=*# show transaction isolation level;
 transaction_isolation
-----------------------
 serializable
(1 row)

  • 在Tx1 中對accounts table進行select query 並進行Sum 和Insert

    -- Tx1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
  7 | sum  |     270 | USD      | 2023-08-23 12:15:16.939807+00
(5 rows)

simple_bank=*# select sum(balance) from accounts;
 sum
-----
 810
(1 row)

simple_bank=*# insert into accounts(owner, balance, currency) values ('sum2', 810, 'USD') returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
 10 | sum  |     810 | USD      | 2023-08-23 12:19:23.223866+00
(1 row)

INSERT 0 1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
  7 | sum   |     270 | USD      | 2023-08-23 12:15:16.939807+00
 10 | sum   |     810 | USD      | 2023-08-23 12:19:23.223866+00
(6 rows)

  • 在Tx2 中對accounts table進行select query 並進行Sum 和Insert

    simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
  7 | sum   |     270 | USD      | 2023-08-23 12:15:16.939807+00
(5 rows)

simple_bank=*# select sum(balance) from accounts;
 sum
-----
 810
(1 row)

simple_bank=*# insert into accounts(owner, balance, currency) values ('sum3', 810, 'USD') returning *;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
 11 | sum  |     810 | USD      | 2023-08-23 12:17:07.013085+00
(1 row)

INSERT 0 1
simple_bank=*# select * from accounts;
 id | owner | balance | currency |          created_at
----+-------+---------+----------+-------------------------------
  2 | two   |     100 | USD      | 2023-08-23 11:03:23.441965+00
  3 | three |     100 | USD      | 2023-08-23 11:03:23.441965+00
  1 | one   |      70 | USD      | 2023-08-23 11:03:23.441965+00
  4 | sum   |     270 | USD      | 2023-08-23 11:46:02.890059+00
  7 | sum   |     270 | USD      | 2023-08-23 12:15:16.939807+00
 11 | sum   |     810 | USD      | 2023-08-23 12:17:07.013085+00
(6 rows)

  • 並對Tx1 和 Tx2 進行Commit

    • Tx1成功提交。
    • Tx2因交易之間的讀/寫依賴關係而拋出錯誤。
    • Postgres建議重試可能會使交易成功。
    • concurrent的交易不再創建重複的記錄。
    • Postgres使用dependencies detection機制來防止serializable anomaly。
    -- Tx1:
simple_bank=# commit;
COMMIT

-- Tx2:
simple_bank> commit;
ERROR:  could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during commit attempt.
HINT:  The transaction might succeed if retried.

上一篇
[Day 13] Understand isolation levels & read phenomena in MySQL
下一篇
[Day 15] Summary about relationship between isolation levels and read phenomena
系列文
Techschool Goalng Backend Master Class 的學習記錄 共 31 篇
  1. 27
    [Day 27] Add users table with unique & foreign key constraints in PostgreSQL
  2. 28
    [Day 28] How to handle DB errors in Golang correctly
  3. 29
    [Day 29] How to securely store passwords?
  4. 30
    [Day 30] How to write stronger unit tests with a custom go-mock matcher
  5. 31
    [Day 31] Why PASETO is better than JWT for token-based authentication?
完整目錄
圖片
  熱門推薦
圖片
{{ item.channelVendor }} | {{ item.webinarstarted }} |
{{ formatDate(item.duration) }}
直播中

尚未有邦友留言

立即登入留言
iThome鐵人賽
參賽組數
902 組
團體組數
37 組
累計文章數
19772 篇
完賽人數
529 人
iT+看影片追技術 看影片追技術 看更多
熱門tag 看更多
15th鐵人賽 16th鐵人賽 13th鐵人賽 14th鐵人賽 17th鐵人賽 12th鐵人賽 11th鐵人賽 鐵人賽 2019鐵人賽 javascript 2018鐵人賽 python 2017鐵人賽 windows php c# linux windows server css react
熱門問題
熱門回答
熱門文章
IT邦幫忙